Systems and methods for processing images to determine image-based computational biomarkers from liquid specimens

ABSTRACT

A method of using a machine learning model to output a task-specific prediction may include receiving a digitized cytology image of a cytology sample and applying a machine learning model to isolate cells of the digitized cytology image. The machine learning model may include identifying a plurality of sub-portions of the digitized cytology image, identifying, for each sub-portion of the plurality of sub-portions, either background or cell, and determining cell sub-images of the digitized cytology image. Each cell sub-image may comprise a cell of the digitized cytology image, based on the identifying either background or cell. The method may further comprise determining a plurality of features based on the cell sub-images, each of the cell sub-images being associated with at least one of the plurality of features, determining an aggregated feature based on the plurality of features, and training a machine learning model to predict a target task based on the aggregated feature.

RELATED APPLICATION(S)

This application claims priority to U.S. Provisional Application No.63/107,389 filed Oct. 29, 2020, the entire disclosure of which is herebyincorporated herein by reference in its entirety.

FIELD OF THE DISCLOSURE

Various embodiments of the present disclosure pertain generally to imageprocessing methods. More specifically, particular embodiments of thepresent disclosure relate to systems and methods for processingelectronic images and rapid evaluation of adequacy in specimenpreparations.

BACKGROUND

In the field of cancer and other disease diagnosis, diagnostic andscreening tests are used to detect disease. Typically, diagnostic testsare conducted if a screening test result is positive. Cytotechniciansare laboratory professionals who review slides of human cells todetermine if they are abnormal or diseased. Their initial findings areused by pathologists to render a diagnosis for the disease. Based on thediagnosis, pathologists and physicians can request additional moleculartesting (e.g. flow, IHC, FISH, PCR/NGS) to better segment the besttreatment pathway for the patient. However, manual or human review ofslides can involve disadvantages based on speed, cost, and accuracy.

Cytology tests are a mainstay of malignancy screening because they areeasier to acquire, less invasive, less expensive, and result in lesscomplications, compared to manual or human review of tissue slides. Theycan also be prepared more quickly for pathology diagnosis when comparedto tissue samples. They are, however, challenging to diagnose for amyriad of reasons, including: 1) ancillary tests such asimmunohistochemistry might not be available, and 2) samples are oftenscant making a specific diagnosis more difficult. As such, cytologysamples are often classified as ‘adequate to make a diagnosis’ or‘inadequate to make a diagnosis’ and then, if adequate, simplycategorized (rather than specifically diagnosed) as benign, atypical,suspicious for malignancy, or positive for malignancy. Depending on theresults of cytology, a confirmatory tissue sample can be performed, ordefinitive treatment can be undertaken. In addition to diagnosticinformation, information about the molecular makeup of the tumor can beobtained from both cytology and tissue samples, providing importantguidance on possible non-surgical treatment options. This geneticinformation however, necessitates enough tumor tissue to be able totest.

The background description provided herein is for the purpose ofgenerally presenting the context of the disclosure. Unless otherwiseindicated herein, the materials described in this section are not priorart to the claims in this application and are not admitted to be priorart, or suggestions of the prior art, by inclusion in this section.

SUMMARY

According to certain aspects of the present disclosure, systems andmethods are disclosed for using a machine learning model to output atask-specific prediction.

A method of using a machine learning model to output a task-specificprediction may include receiving a digitized cytology image of acytology sample and applying a machine learning model to isolate cellsof the digitized cytology image. The machine learning model may includeidentifying a plurality of sub-portions of the digitized cytology image,identifying, for each sub-portion of the plurality of sub-portions,either background or cell, and determining cell sub-images of thedigitized cytology image. Each cell sub-image may comprise a cell of thedigitized cytology image, based on the identifying either background orcell. The method of using the machine learning model may furthercomprise determining a plurality of features based on the cellsub-images, each of the cell sub-images being associated with at leastone of the plurality of features, determining an aggregated featurebased on the plurality of features, and training a machine learningmodel to predict a target task based on the aggregated feature.

Identifying either background or cell may comprise using a segmentationsystem and/or a detection system. Using the segmentation system maycomprise identifying one or more individual pixels in the digitizedcytology image as belonging to a cell or background, classifying one ormore cell regions, of a cell of interest, containing the one or moreindividual pixels into at least one granular structure, and outputtingthe cell sub-images tightly bounded around the cell of interest withinthe digitized cytology image.

Identifying either background or cell may comprise using a detectionsystem to identify bounding regions for each of the cell sub-images.

The method may further comprise using the aggregated feature to train amachine learning classifier to predict necessary quantifications for atarget task on a per-cell level. Determining the aggregated feature maycomprise computing statistics of per-cell classifications, estimatingfeature means or cluster centers, and/or training deep learningaggregators that combine per-cell features through convolutional orrecurrent mechanisms.

The target task may comprise ensuring the cytology sample containsenough material for a user to categorize the cytology sample. The targettask may comprise classifying a specimen category from the aggregatedfeature. The target task may comprise predicting a probability ofdetecting a specific mutation within a cytology specimen.

A system for using a machine learning model to output a task-specificprediction may comprise at least one memory storing instructions and atleast one processor configured to execute the instructions to performoperations. The operations performed by the processor may comprisereceiving a digitized cytology image of a cytology sample and applying amachine learning model to isolate cells of the digitized cytology image.Applying the machine learning model may comprise identifying a pluralityof sub-portions of the digitized cytology image, identifying, for eachsub-portion of the plurality of sub-portions, either background or cell,and determining cell sub-images of the digitized cytology image. Eachcell sub-image may comprise a cell of the digitized cytology image,based on the identifying either background or cell. The operationsperformed by the processor may further comprise determining a pluralityof features based on the cell sub-images, each of the cell sub-imagesbeing associated with at least one of the plurality of features,determining an aggregated feature based on the plurality of features,and training a machine learning model to predict a target task based onthe aggregated feature.

Identifying either background or cell my comprise using a segmentationsystem and/or a detection system. Using the segmentation system maycomprise identifying one or more individual pixels in the digitizedcytology image as belonging to a cell or background, classifying one ormore cell regions, of a cell of interest, containing the one or moreindividual pixels into at least one granular structure, and outputtingthe cell sub-images tightly bounded around the cell of interest withinthe digitized cytology image.

Identifying either background or cell may comprise using a detectionsystem to identify bounding regions for each of the cell sub-images.

The operations may further comprise using the aggregated feature totrain a machine learning classifier to predict necessary quantificationsfor a target task on a per-cell level. Determining the aggregatedfeature may comprise computing statistics of per-cell classifications,estimating feature means or cluster centers, and/or training deeplearning aggregators that combine per-cell features throughconvolutional or recurrent mechanisms.

The target task may comprise ensuring the cytology sample containsenough material for a user to categorize the cytology sample. The targettask may comprise classifying a specimen category from the aggregatedfeature. The target task may comprise predicting a probability ofdetecting a specific mutation within a cytology specimen.

A non-transitory computer-readable medium may store instructions that,when executed by a processor, perform a method of using a machinelearning model to output a task-specific prediction. The method maycomprise receiving a digitized cytology image of a cytology sample andapplying a machine learning model to isolate cells of the digitizedcytology image. Applying the machine learning model may includeidentifying a plurality of sub-portions of the digitized cytology image,identifying, for each sub-portion of the plurality of sub-portions,either background or cell, and determining cell sub-images of thedigitized cytology image. Each cell sub-image may comprise a cell of thedigitized cytology image, based on the identifying either background orcell. The method of using the machine learning model may includedetermining a plurality of features based on the cell sub-images, eachof the cell sub-images being associated with at least one of theplurality of features, determining an aggregated feature based on theplurality of features, and training a machine learning model to predicta target task based on the aggregated feature. Identifying eitherbackground or cell may comprise using a segmentation system and/or adetection system.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory only,and are not restrictive of the disclosed embodiments.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in, and constitute apart of, this specification, illustrate various exemplary embodimentsand together with the description, serve to explain the principles ofthe disclosed embodiments.

FIG. 1A illustrates an exemplary block diagram of a system and networkto detect cellular features from a cytology specimen, according to anexemplary embodiment of the present disclosure.

FIG. 1B illustrates an exemplary block diagram of a disease detectionplatform, according to an exemplary embodiment of the presentdisclosure.

FIG. 1C illustrates an exemplary block diagram of a slide analysis tool,according to an exemplary embodiment of the present disclosure.

FIG. 2 is a flowchart illustrating an exemplary method for predicting atarget task from a cytology specimen, according to an exemplaryembodiment of the present disclosure.

FIG. 3 is a flowchart illustrating an exemplary method for evaluating anadequacy of non-malignant cells in a pathology specimen, according to anexemplary embodiment of the present disclosure

FIG. 4 illustrates an example system that may execute techniquespresented herein.

DESCRIPTION OF THE EMBODIMENTS

Reference will now be made in detail to the exemplary embodiments of thepresent disclosure, examples of which are illustrated in theaccompanying drawings. Wherever possible, the same reference numberswill be used throughout the drawings to refer to the same or like parts.

The systems, devices, and methods disclosed herein are described indetail by way of examples and with reference to the figures. Theexamples discussed herein are examples only and are provided to assistin the explanation of the apparatuses, devices, systems, and methodsdescribed herein. None of the features or components shown in thedrawings or discussed below should be taken as mandatory for anyspecific implementation of any of these devices, systems, or methodsunless specifically designated as mandatory.

Also, for any methods described, regardless of whether the method isdescribed in conjunction with a flow diagram, it should be understoodthat unless otherwise specified or required by context, any explicit orimplicit ordering of steps performed in the execution of a method doesnot imply that those steps must be performed in the order presented butinstead may be performed in a different order or in parallel.

As used herein, the term “exemplary” is used in the sense of “example,”rather than “ideal.” Moreover, the terms “a” and “an” herein do notdenote a limitation of quantity, but rather denote the presence of oneor more of the referenced items.

In the spirit of less invasive screening tests, cytology suffers fromlimited tumor quantity and an inability to use common ancillary testingfor diagnosis refinement, both of which result in less specificdiagnostic categorization. Furthermore, as molecular evaluation of thetumor becomes mainstay, the limited sampling performed in cytology mayhinder a comprehensive molecular study of the tumor.

Current industry practice includes a patient undergoing cytologicsampling as part of a screening test or limited, less invasive samplingof a suspected malignancy. The pathologist receives the cytologic smearsand preparations, and renders a categorical diagnosis of benign,atypical, suspicious for malignancy, or positive for malignancy, afterassessing the specimen for adequacy. Occasionally ancillary testing canbe performed which consumes previous tumor tissue, which may otherwisebe available for optional additional molecular studies.

The nature of tumor preparation in cytology limits a specific diagnosis;limited material can hinder diagnosis and ability to perform ancillarystudies as well as time to diagnose specimen and cost.

The systems and methods of this disclosure leverage artificialintelligence (AI) technology to detect cellular features that arenecessary for pathological diagnosis, prognostic and treatmentdecisions. Data and predictions are aggregated and made availableinstantaneously via any user interface (e.g. through a digital pathologyviewing system, report, or laboratory information system, etc.). Machinelearning algorithms can rapidly and simultaneously assess for adequacy,categorize the sample into diagnostic categories, and screen for themost likely molecular changes, limiting the total molecular testingperformed on a tumor and therefore, increasing the likelihood of a validmolecular result due to sufficient quantities of tumor

Thus, the embodiments of this disclosure may allow for rapid evaluationof ‘adequacy’ in liquid-based tumor preparations; facilitate thediagnosis of liquid based tumor preparations (cytology,hematology/hematopathology); and predict molecular findings most likelyto be found in various tumors detected by liquid-based preparations

This may enable more accurate diagnosis of tumors on liquid-basedpreparations, and it may enable physicians to identify patients that maybenefit from molecular testing and/or a corresponding treatment sooner.

FIG. 1A illustrates an exemplary block diagram of a system and networkto detect cellular features from a cytology sample, according to anexemplary embodiment of the present disclosure.

Specifically, FIG. 1A illustrates an electronic network 120 that may beconnected to servers at hospitals, laboratories, and/or doctor'soffices, etc. For example, physician servers 121, hospital servers 122,clinical trial servers 123, research lab servers 124, and/or laboratoryinformation systems 125, etc., may each be connected to an electronicnetwork 120, such as the Internet, through one or more computers,servers and/or handheld mobile devices. According to an exemplaryembodiment of the present application, the electronic network 120 mayalso be connected to server systems 110, which may include processingdevices that are configured to implement a disease detection platform100, which includes a slide analysis tool 101 for determining specimenproperty or image property information pertaining to digital pathologyimage(s), and using machine learning to determine whether a disease orinfectious agent is present, according to an exemplary embodiment of thepresent disclosure. The slide analysis tool 101 may allow for rapidevaluation of ‘adequacy’ in liquid-based tumor preparations; facilitatethe diagnosis of liquid based tumor preparations (cytology,hematology/hematopathology); and predict molecular findings most likelyto be found in various tumors detected by liquid-based preparations.

The physician servers 121, hospital servers 122, clinical trial servers123, research lab servers 124 and/or laboratory information systems 125may create or otherwise obtain images of one or more patients' cytologyspecimen(s), histopathology specimen(s), slide(s) of the cytologyspecimen(s), digitized images of the slide(s) of the histopathologyspecimen(s), or any combination thereof. The physician servers 121,hospital servers 122, clinical trial servers 123, research lab servers124 and/or laboratory information systems 125 may also obtain anycombination of patient-specific information, such as age, medicalhistory, cancer treatment history, family history, past biopsy orcytology information, etc. The physician servers 121, hospital servers122, clinical trial servers 123, research lab servers 124 and/orlaboratory information systems 125 may transmit digitized slide imagesand/or patient-specific information to server systems 110 over theelectronic network 120. Server system(s) 110 may include one or morestorage devices 109 for storing images and data received from at leastone of the physician servers 121, hospital servers 122, clinical trialservers 123, research lab servers 124, and/or laboratory informationsystems 125. Server systems 110 may also include processing devices forprocessing images and data stored in the storage devices 109. Serversystems 110 may further include one or more machine learning tool(s) orcapabilities. For example, the processing devices may include a machinelearning tool for a disease detection platform 100, according to oneembodiment. Alternatively or in addition, the present disclosure (orportions of the system and methods of the present disclosure) may beperformed on a local processing device (e.g., a laptop).

The physician servers 121, hospital servers 122, clinical trial servers123, research lab servers 124 and/or laboratory information systems 125refer to systems used by pathologists for reviewing the images of theslides. In hospital settings, tissue type information may be stored in alaboratory information system 125.

FIG. 1B illustrates an exemplary block diagram of a disease detectionplatform 100 for determining specimen property or image propertyinformation pertaining to digital pathology image(s), using machinelearning. The disease detection platform 100 may include a slideanalysis tool 101, a data ingestion tool 102, a slide intake tool 103, aslide scanner 104, a slide manager 105, a storage 106, a laboratoryinformation system 107 and a viewing application tool 108.

The slide analysis tool 101, as described below, refers to a process andsystem for determining data variable property or health variableproperty information pertaining to digital pathology image(s). Machinelearning may be used to classify an image, according to an exemplaryembodiment. The slide analysis tool 101 may also predict futurerelationships, as described in the embodiments below.

The data ingestion tool 102 may facilitate a transfer of the digitalpathology images to the various tools, modules, components, and devicesthat are used for classifying and processing the digital pathologyimages, according to an exemplary embodiment.

The slide intake tool 103 may scan pathology images and convert theminto a digital form, according to an exemplary embodiment. The slidesmay be scanned with slide scanner 104, and the slide manager 105 mayprocess the images on the slides into digitized pathology images andstore the digitized images in storage 106.

The viewing application tool 108 may provide a user with a specimenproperty or image property information pertaining to digital pathologyimage(s), according to an exemplary embodiment. The information may beprovided through various output interfaces (e.g., a screen, a monitor, astorage device and/or a web browser, etc.).

The slide analysis tool 101, and one or more of its components, maytransmit and/or receive digitized slide images and/or patientinformation to server systems 110, physician servers 121, hospitalservers 122, clinical trial servers 123, research lab servers 124,and/or laboratory information systems 125 over a network 120. Further,server systems 110 may include storage devices for storing images anddata received from at least one of the slide analysis tool 101, the dataingestion tool 102, the slide intake tool 103, the slide scanner 104,the slide manager 105, and viewing application tool 108. Server systems110 may also include processing devices for processing images and datastored in the storage devices. Server systems 110 may further includeone or more machine learning tool(s) or capabilities, e.g., due to theprocessing devices. Alternatively, or in addition, the presentdisclosure (or portions of the system and methods of the presentdisclosure) may be performed on a local processing device (e.g., alaptop).

Any of the above devices, tools and modules may be located on a devicethat may be connected to an electronic network such as the Internet or acloud service provider, through one or more computers, servers and/orhandheld mobile devices.

FIG. 1C illustrates an exemplary block diagram of a slide analysis tool101, according to an exemplary embodiment of the present disclosure. Theslide analysis tool 101 may include a training image platform 131 and/ora target image platform 135.

According to one embodiment, the training image platform 131 may includea training image intake module 132, a data analysis module 133, and acell identification module 134.

The training image platform 131, according to one embodiment, may createor receive training images that are used to train a machine learningmodel to effectively analyze and classify digital pathology images. Forexample, the training images may be received from any one or anycombination of the server systems 110, physician servers 121, hospitalservers 122, clinical trial servers 123, research lab servers 124,and/or laboratory information systems 125. Images used for training maycome from real sources (e.g., humans, animals, etc.) or may come fromsynthetic sources (e.g., graphics rendering engines, 3D models, etc.).Examples of digital pathology images may include (a) digitized slidesstained with a variety of stains, such as (but not limited to) H&E,Hematoxylin alone, IHC, molecular pathology, etc.; and/or (b) digitizedtissue samples from a 3D imaging device, such as microCT.

The training image intake module 132 may create or receive a datasetcomprising one or more training datasets corresponding to one or morehealth variables and/or one or more data variables. For example, thetraining datasets may be received from any one or any combination of theserver systems 110, physician servers 121, hospital servers 122,clinical trial servers 123, research lab servers 124, and/or laboratoryinformation systems 125. This dataset may be kept on a digital storagedevice. The data analysis module 133 may identify whether a set ofindividual cells belong to a cell of interest or a background of adigitized image. The cell identification module 134 may analyzedigitized images and determine whether an individual cell in thecytology sample needs further analysis. It is useful to identify whetheran individual cell needs further analysis and to aggregate these areas,and the identification of such may trigger an alert to a user.

According to one embodiment, the target image platform 135 may include atarget image intake module 136, a specimen detection module 137, and anoutput interface 138. The target image platform 135 may receive a targetimage and apply the machine learning model to the received target imageto determine a characteristic of a target data set. For example, thetarget data may be received from any one or any combination of theserver systems 110, physician servers 121, hospital servers 122,clinical trial servers 123, research lab servers 124, and/or laboratoryinformation systems 125. The target image intake module 136 may receivea target dataset corresponding to a target health variable or a datavariable. Specimen detection module 137 may apply the machine learningmodel to the target dataset to determine a characteristic of the targethealth variable or a data variable. For example, the specimen detectionmodule 137 may detect a trend of the target relationship. The specimendetection module 137 may also apply the machine learning model to thetarget dataset to determine a quality score for the target dataset.Further, the specimen detection module 137 may apply the machinelearning model to the target images to determine whether a targetelement is present in a determined relationship.

The output interface 138 may be used to output information about thetarget data and the determined relationship. (e.g., to a screen,monitor, storage device, web browser, etc.).

FIG. 2 is a flowchart illustrating an exemplary method for predicting atarget task from a cytology specimen, according to an exemplaryembodiment of the present disclosure. For example, a target task 200 maybe to assess adequacy of the specimen using cellularity by identifyingindividual cells (e.g., steps 202-210), then counting the number ofnon-malignant cells using a machine learning method 212, which may beperformed by slide analysis tool 101 automatically or in response to arequest from a user.

According to an exemplary embodiment 200, the artificial intelligence(AI) detection system may be comprised of three components: cellisolation, cell feature extraction, and output aggregation. Cellisolation extracts individual cell patches from the cytology image,feature extraction identifies important morphological features from thecell, and output aggregation combines all the cellular features into afinal output prediction depending on the task. The input to the systemmay be a digital cytology image, and the output may be a task-specificprediction including but not limited to diagnosis, prognosis, ortreatment decision. These three components may include one or more ofthe following steps.

In step 202, the method may include receiving a digital cytology imageinto a digital storage device.

In step 204, the method may include extracting at least one cell patchfrom the digitized cytology image.

In step 206, the method may include identifying the at least one cellpatch as belonging to either a cell of interest or to a background ofthe digitized cytology image.

In step 208, the method may include applying a machine learning model toisolate a set of individual cells within the cell patch from thebackground material of the digitized cytology image. The process of cellisolation identifies individual cells within the cytology image,separating the cells of interest from the background material within thefluid. Isolating each cell may be accomplished by a machine learningmodel trained to identify the cell type of interest. The machinelearning model can take the form of a detection or segmentation system.The detection approach identifies bounding regions for each cell and canbe accomplished by a variety of methods including but not limited to:

-   -   a) Sliding-window approaches using region features such as Haar        wavelets or histogram of oriented gradients (HOG) and a        classifier such as boosting, support vector machine (SVM), or        deformable parts models (DPM).    -   b) Region proposal-based methods using deep learning such as        region-based convolutional neural networks (R-CNN), spatial        pyramid pooling (SPP-net), you only look once (YOLO) and single        shot detector (SSD).

Segmentation identifies individual pixels as belonging to the cell ofinterest or background. Segmentation can also classify cell regions intomore granular structures such as identifying the nucleus, cytoplasm, andother sub-cellular features. Methods for segmentation can include but isnot limited to:

-   -   a) Edge-based segmentation, histogram-based segmentation, region        growing, graph partitioning and watershed methods.    -   b) Deep learning methods such as UNet, DeepLab and other fully        convolutional network (FCN) variants.

Detection and segmentation can also be combined into a panoptic instancesegmentation system using, but not limited to, the following methods:

-   -   a) Mask R-CNN    -   b) Deep Mask    -   c) PolyTransform    -   d) Detection Transformers (DETR)

The output of the cell isolation system may be a set of sub-imagestightly bounded around each cell of interest within the cytology image.

In step 208, the method may include applying a machine learning model toisolate a set of individual cells within the cell patch from abackground material of the digitized cytology image.

In step 210, the method may include analyzing the isolated individualcells to extract at least one feature that is representative forpredicting a target task.

Each isolated cell may be analyzed by the machine learning model toextract features that are representative for predicting the target task.Methods for extracting these features can include, but are not limitedto:

-   -   1. Color descriptors, shape features, texture descriptors    -   2. Features trained from deep learning models

In step 212, the method may include applying a machine learningclassifier to the at least one extracted feature of the isolatedindividual cells to predict necessary quantifications for the targettask on a per-cell level.

These features may be used to train a machine learning classifier topredict necessary quantifications for the target task on a per-celllevel. These quantifications may include identifying the presence ofcancer, genetic mutation, hypertrophy, etc.

In step 214, the method may include combining the at least one extractedfeature of the individual cells of interest into an aggregate assessmentto represent the digitized cytology image as a whole. This aggregationcan be achieved by a variety of approaches, including computingstatistics of the per-cell classifications, estimating feature means orcluster centers, and training deep learning aggregators that combineper-cell features through convolutional or recurrent mechanisms.

Once each cell is classified, the results for all cells in the cytologyimage are combined into an aggregate assessment to achieve a finalprediction of the target task.

In step 216, the method may include applying a machine learning model tothe aggregate assessment to predict the target task. The finalaggregated feature may also be used to train a final machine learningmodel to predict the target task. This can include methods such as SVM,logistic regression, and regression or classification-based neuralnetworks.

Using a Machine Learning Model to Predict the Target Task

According to an exemplary embodiment, one or more steps for performing amethod may include the following steps. All steps are optional and maybe used in any order.

-   -   1. Receive digitized liquid sample.    -   2. Optionally receive meta data (e.g. patient clinical        information)    -   3. An AI-based system may be used to generate a result    -   4. The prediction may be outputted to an electronic storage        device and/or display.    -   5. The user (e.g. Pathologist, oncologist, patient, etc.) may be        made aware that results are available, for example via push or        pull notification, user interface popup, a noise, etc.    -   6. Optionally, the user may be provided the option to review        visualization and/or report. This may also occur automatically.

The present disclosure may be implemented in the following ways for theuser, including but not limited to:

-   -   1. Within the clinical workflow at the hospital, lab, medical        center as a:        -   a. Web application (cloud-based or on-premises)        -   b. Mobile application        -   c. Interactive report        -   d. Static report        -   e. Dashboard    -   2. Multiple features can be visualized for a single whole slide        image.

The technical workflow behind an embodiment of the present disclosuremay be as follows:

-   -   1. Digitized whole slide image may be created and some or all        metadata may be made available from hospital and/or hardware        databases    -   2. Image and corresponding data may be passed into the        appropriate AI-based system and outputs are generated    -   3. Some of the outputs may be fed into the system that will        generate and display the visualization to the user.        Visualization to the user can be in the form of a:        -   a. Report        -   b. Overlay in a web application, and/or        -   c. Exportable file.

The method above may have many possible specific applications orembodiments, as described below. All steps are optional and may be usedin any order.

Evaluation of Adequacy of Non-Malignant Cells

Various classification/categorization systems have definitions ofadequacy to ensure that each sample contains enough material for theuser to definitively categorize the sample. An exemplary embodiment ofthe present disclosure for this application may be as follows.

FIG. 3 is a flowchart of an exemplary method for evaluating the adequacyof non-malignant cells, according to an exemplary embodiment of thepresent disclosure. For example, exemplary methods 300 and 320 (e.g.,steps 302-310 and steps 322-328) may be performed by the slide analysistool 101 automatically or in response to a request by a user (e.g., aphysician).

In step 302, the method may include receiving one or more digital imagesof a cytology specimen into a digital storage device (e.g. hard drive,network drive, cloud storage, system memory, etc.).

In step 304, the method may include receiving a collection of labels fora set of individual cells in the cytology image indicating malignancyand/or a count of malignant cells for the cytology image.

In step 306, the method may include training a machine learning model toisolate an individual cell in the cytology image, as described above andin FIG. 2 .

In step 308, the method may include extracting features from the set ofindividual cells, as described above and in FIG. 2 .

In step 310, the method may include training a machine learning model toestimate a number of malignant cells. If ground-truth labels areprovided to determine if each cell is malignant or non-malignant, amodel can be trained in a fully-supervised manner using the cellfeatures as input. The model can consist of an SVM, random forest, CNN,etc.

If only a malignant cell count is available, an aggregation network canbe trained to regress the malignant cell count from the collection ofcell features. This can utilize neural network models such as arecurrent neural network (RNN) or convolutional neural network (CNN).

In step 322, the method may include receiving one or more digital imagesof a cytology specimen into a digital storage device (e.g. hard drive,network drive, cloud storage, system memory, etc.).

In step 324, the method may include applying the machine learning modelto isolate and extract at least one feature from each cell.

In step 326, the method may include classifying each cell as malignantor non-malignant.

In step 328, the method may include determining a count or amount ofmalignant cells. If a fully-supervised model was trained to classifymalignant cells, malignant cells can be counted to determine adequacy.Otherwise, the regression model can be used to directly estimate thecount.

Categorization of Specimen

Various classification/categorization systems may be used for certaincytologic samples (e.g. Bethesda for cervical/Pap and thyroid samples,Paris for urine samples, Milan for salivary samples). Based on themorphological features present on the slide, the present disclosureidentifies the specimen type and selects the appropriate categorizationsystem for classifying the findings in the specimen.

For Training:

-   -   1. One or more digital images of a cytology specimen may be        received at a digital storage device or processor (e.g. hard        drive, network drive, cloud storage, system memory, etc.).    -   2. A label for each cytology image may be received indicating        the categorization system, and the category of the specimen.    -   3. A machine learning algorithm may be trained to classify the        specimen type. This may be learned directly from the cytology        image using algorithms such as CNN, random forest, or SVM. This        may also be learned by cell isolation and feature aggregation as        described in the general form of the present disclosure.        Determination of the specimen type may also determine the        categorization system.    -   4. The cells may be isolated and cellular features may be        extracted as described in the general form of the present        disclosure.    -   5. The output aggregation model may be trained to combine cells        features to predict the category using the target labels        specified in step 2.

For Use:

-   -   1. One or more digital images of a cytology specimen may be        received a digital storage device and/or display (e.g. hard        drive, network drive, cloud storage, system memory, etc.).    -   2. The specimen type may be classified using the trained        specimen classification model.    -   3. The cells and may be isolated and cellular features may be        extracted as described in the general form of the present        disclosure.    -   4. The specimen category may be classified from the cellular        features using the trained output aggregation model.

Screening for Mutations

Specific cancers can exhibit mutations detected by fluorescence in situhybridization (FISH) or next-generation sequencing (NGS). Thesemutations may be prognostic or have therapeutic implications. Thepresent disclosure may allow an algorithm to predict the probability ofdetecting specific mutations in tumor cells within a cytologicpreparation by FISH or NGS.

For Training:

-   -   1. One or more digital images of a cytology specimen may be        received at a processor and/or digital storage device (e.g. hard        drive, network drive, cloud storage, system memory, etc.).    -   2. Labels for individual cells may be received in the cytology        image indicating if they are malignant or not.    -   3. Labels for individual cells in the cytology image may        originate from spatial genomics to identify specific gene        mutations or biomarkers for each cell that correlate with        malignancy; e.g. EGFR, ALK, KRAS, ROS1, RET, etc. for lung,        BRAF, NRAS, RET, etc. for thyroid, IDH1, IDH2, FLT3, NPM1, 17p-,        etc. for bone marrow aspirates, etc.    -   4. A machine learning algorithm may be trained to isolate each        cell as described in the general form of the present disclosure.    -   5. Features may be extracted from each cell as described in the        general form of the present disclosure.    -   6. A machine learning algorithm may be trained to classify if a        cell is malignant or not using the cellular features extracted        in the previous step and the target labels in step 2. The model        may consist of an SVM, random forest, CNN, etc.    -   7. The above model may be used to isolate only the cells that        are malignant and train a machine learning algorithm to classify        the presence of the mutation in that cell using the target        mutation labels in step 3. The model can consist of an SVM,        random forest, CNN, etc.

For Use:

-   -   1. One or more digital images may be received of a cytology        specimen into a digital storage device (e.g. hard drive, network        drive, cloud storage, system memory, etc.).    -   2. The machine learning algorithm may be applied to isolate and        extract features from each cell.    -   3. Each cell may be classified as malignant or non-malignant.    -   4. The malignant cells may be isolated. The presence of any        mutation and a corresponding classification score may be        predicted.

As shown in FIG. 4 , device 400 may include a central processing unit(CPU) 420. CPU 420 may be any type of processor device including, forexample, any type of special purpose or a general-purpose microprocessordevice. As will be appreciated by persons skilled in the relevant art,CPU 420 also may be a single processor in a multi-core/multiprocessorsystem, such system operating alone, or in a cluster of computingdevices operating in a cluster or server farm. CPU 420 may be connectedto a data communication infrastructure 410, for example a bus, messagequeue, network, or multi-core message-passing scheme.

Device 400 may also include a main memory 440, for example, randomaccess memory (RAM), and also may include a secondary memory 430.Secondary memory 430, e.g. a read-only memory (ROM), may be, forexample, a hard disk drive or a removable storage drive. Such aremovable storage drive may comprise, for example, a floppy disk drive,a magnetic tape drive, an optical disk drive, a flash memory, or thelike. The removable storage drive in this example reads from and/orwrites to a removable storage unit in a well-known manner. The removablestorage may comprise a floppy disk, magnetic tape, optical disk, etc.,which is read by and written to by the removable storage drive. As willbe appreciated by persons skilled in the relevant art, such a removablestorage unit generally includes a computer usable storage medium havingstored therein computer software and/or data.

In alternative implementations, secondary memory 430 may include similarmeans for allowing computer programs or other instructions to be loadedinto device 400. Examples of such means may include a program cartridgeand cartridge interface (such as that found in video game devices), aremovable memory chip (such as an EPROM or PROM) and associated socket,and other removable storage units and interfaces, which allow softwareand data to be transferred from a removable storage unit to device 400.

Device 400 also may include a communications interface (“COM”) 460.Communications interface 460 allows software and data to be transferredbetween device 400 and external devices. Communications interface 460may include a modem, a network interface (such as an Ethernet card), acommunications port, a PCMCIA slot and card, or the like. Software anddata transferred via communications interface 460 may be in the form ofsignals, which may be electronic, electromagnetic, optical or othersignals capable of being received by communications interface 460. Thesesignals may be provided to communications interface 460 via acommunications path of device 400, which may be implemented using, forexample, wire or cable, fiber optics, a phone line, a cellular phonelink, an RF link or other communications channels.

The hardware elements, operating systems, and programming languages ofsuch equipment are conventional in nature, and it is presumed that thoseskilled in the art are adequately familiar therewith. Device 400 mayalso include input and output ports 450 to connect with input and outputdevices such as keyboards, mice, touchscreens, monitors, displays, etc.Of course, the various server functions may be implemented in adistributed fashion on a number of similar platforms, to distribute theprocessing load. Alternatively, the servers may be implemented byappropriate programming of one computer hardware platform.

Throughout this disclosure, references to components or modulesgenerally refer to items that logically can be grouped together toperform a function or group of related functions. Like referencenumerals are generally intended to refer to the same or similarcomponents. Components and modules may be implemented in software,hardware or a combination of software and hardware.

The tools, modules, and functions described above may be performed byone or more processors. “Storage” type media may include any or all ofthe tangible memory of the computers, processors, or the like, orassociated modules thereof, such as various semiconductor memories, tapedrives, disk drives and the like, which may provide non-transitorystorage at any time for software programming.

Software may be communicated through the Internet, a cloud serviceprovider, or other telecommunication networks. For example,communications may enable loading software from one computer orprocessor into another. As used herein, unless restricted tonon-transitory, tangible “storage” media, terms such as computer ormachine “readable medium” refer to any medium that participates inproviding instructions to a processor for execution.

The foregoing general description is exemplary and explanatory only, andnot restrictive of the disclosure. Other embodiments of the inventionwill be apparent to those skilled in the art from consideration of thespecification and practice of the invention disclosed herein. It isintended that the specification and examples to be considered asexemplary only.

1-20. (canceled)
 21. A computer-implemented method for identifying aspecimen category, comprising: receiving one or more digital images of acytology specimen at a digital storage device; isolating one or morecell images using the one or more digital images by isolating a set ofindividual images of cells within the one or more digital images using asegmentation system and/or a detection system; extracting one or morecellular features using the cell images; and determining a specimencategory from the one or more cellular features using a trained outputaggregation model.
 22. The computer-implemented method of claim 21,further comprising training the trained output aggregation model by:receiving one or more training digital images of at least one cytologyspecimen; receiving at least one training label for the cytologyspecimen indicating a categorization system and a category of thecytology specimen; isolating one or more cell images using the one ormore training digital images; and extracting one or more cellularfeatures from the one or more isolated cell images.
 23. Thecomputer-implemented method of claim 22, wherein training the trainedoutput aggregation model includes training the trained outputaggregation model to combine the one or more extracted cellular featuresusing the at least one training label.
 24. The computer-implementedmethod of claim 21, further comprising training at least one of thetrained output aggregation model or a trained specimen classificationmodel to classify a specimen type for a cytology specimen.
 25. Thecomputer-implemented method of claim 24, further comprising determininga specimen type from the received one or more digital images using thetrained output aggregation model or the trained specimen classificationmodel.
 26. The computer-implemented method of claim 21, furthercomprising: extracting at least one cell patch from the one or moredigital images; and identifying the at least one cell patch as belongingto either cell or background.
 27. The computer-implemented method ofclaim 26, wherein the set of individual images of cells are isolatedwithin the cell patch.
 28. The computer-implemented method of claim 21,wherein the segmentation system and/or the detection system is a machinelearning model.
 29. The computer-implemented method of claim 21, furthercomprising: selecting an appropriate categorization system for the oneor more cellular features in the cytology specimen.
 30. Thecomputer-implemented method of claim 21, further comprising combiningthe one or more cellular features in the cytology specimen into anaggregate assessment to represent the digital cytology image as a whole.31. The computer-implemented method of claim 21, further comprising:notifying a user that the specimen category is available; and providingthe user an option to review a visualization and/or a report of thespecimen category.
 32. The computer-implemented method of claim 21,further comprising: outputting the specimen category to a digitalstorage device and/or display.
 33. The computer-implemented method ofclaim 22, wherein outputting the specimen category includes outputting avisualization and/or a report of the specimen category.
 34. A system foridentifying a specimen category, comprising: at least one memory storinginstructions; and at least one processor configured to execute theinstructions to perform operations comprising: receiving one or moredigital images of a cytology specimen at a digital storage device;isolating one or more cell images using the one or more digital imagesby isolating a set of individual images of cells within the one or moredigital images using a segmentation system and/or a detection system;extracting one or more cellular features using the cell images; anddetermining a specimen category from the one or more cellular featuresusing a trained output aggregation model.
 35. The system of claim 34,wherein the operations further comprise training the trained outputaggregation model by: receiving one or more training digital images ofat least one cytology specimen; receiving at least one training labelfor the cytology specimen indicating a categorization system and acategory of the cytology specimen; isolating one or more cell imagesusing the one or more training digital images; and extracting one ormore cellular features from the one or more isolated cell images. 36.The system of claim 35, wherein training the trained output aggregationmodel includes training the trained output aggregation model to combinethe one or more extracted cellular features using the at least onetraining label.
 37. The system of claim 34, wherein the operationsfurther comprise: extracting at least one cell patch from the one ormore digital images; and identifying the at least one cell patch asbelonging to either cell or background, and wherein the set ofindividual images of cells are isolated within the cell patch.
 38. Thesystem of claim 34, further comprising at least one of a display or adigital storage device, wherein the operations further compriseoutputting the specimen category to the at least one of the display orthe digital storage device.
 39. A non-transitory computer readablemedium storing instructions that, when executed by a processor, causethe processor to perform a method for identifying a specimen category,the method comprising: receiving one or more digital images of acytology specimen at a digital storage device; extracting at least onecell patch from the one or more digital images; identifying the at leastone cell patch as belonging to either cell or background; isolating oneor more cell images using the digital images, wherein isolating the oneor more cell images comprises isolating a set of individual images ofcells within the cell patch using a segmentation system and/or adetection system; extracting one or more cellular features using thecell images; and determining a specimen category from the one or morecellular features using a trained output aggregation model.
 40. Thenon-transitory computer readable medium of claim 39, the method furthercomprising: extracting at least one cell patch from the one or moredigital images; and identifying the at least one cell patch as belongingto either cell or background, wherein the set of individual images ofcells are isolated within the cell patch.